Method and system for developing image processing applications

ABSTRACT

The described method relates to the delelopment of image processing applications. In order to facilitate the development of image processing applications the described method provides a plurality of image processing components ( 1  to  10 ) and a connector means ( 11 ) for establishing connections between components selected from said plurality of components by a user, wherein the connector means ( 11 ) automatically connects said selected components. In particular, the automatic connection of the selected components is based on information provided by the respective components.

[0001] Method and system for developing image processing applicationsThe present invention relates to a method and a system for developingimage processing applications and a computer readable program productfor performing such a method.

[0002] Standard approaches for the development of image processingapplications and basic information thereon can, for example, be found in“Visual Basic 6.0 Programmer's Guide”, Microsoft Press, and in U.S. Pat.No. 5,742,504 describing a method and a system for quickly developingapplication software for use in a machine vision system.

[0003] The following features are common to standard approaches known inthe art. Known systems use components (e.g., ActiveX), which solvespecific sub problems (e.g., image grabbing, or blob analysis), andcombine the components to build the application (design mode).Furthermore, adapting the components and their graphical user interface(design mode) is common to known systems. The state of the components,i.e., the things that have been changed in the design mode.

[0004] However, the following limitations are related with these priorart systems. Each component of the prior art systems has at most onegraphical user interface. Furthermore, in known systems two modes aredistinguished: design mode and user mode. In design mode, the componentsare selected, the program is written, and the state of the features ismodified. The user mode corresponds to the final application(executable). The state of the design mode and the state of the usermode can be saved. However, this is done with two independent andincompatible mechanisms. This means, for example, that changes made inthe user mode cannot be accessed in design mode. Furthermore, the stateof the design mode cannot be changed once the application is generatedbecause it is compiled into the application. Moreover, in design mode,only one component can be executed. Finally, only predefined datastructures are used in known systems.

[0005] A typical problem with existing systems that are based on objectcomponents is the need for programming. The user must add code toconnect the selected components, to specify the control flow, and toconnect data sources like images and regions of interest to theprocessing components. In this context, reference is made to U.S. Pat.No. 5,742,504. However, many users, whose principal goal is to developan image processing application, do not have sufficient programmingknowledge to accomplish the required programming task easily, quickly,and without programming errors.

[0006] It is the object of the present invention to ease the task ofapplication development. This object is solved with the features of theclaims.

[0007] It is an advantage of the present invention that the need forprogramming, the specification of component relations, the specificationof execution order, and the need to store the program status by the userare eliminated. According to the invention, all these tasks aredetermined automatically without any additional task that the user mustperform. The invention is based on components which can be selected bythe user. Components are objects in the terminology of object orientedprogramming. The selection of components is done according to therequirements of the application, i.e., the user selects those subtasksthat fulfill the requirements. Triggered by this selection, according tothe invention, the components automatically generate the application.The user only needs to configure the look and feel of the graphical userinterface and to specify the image processing parameters. By this,programming is completely eliminated from application development.Furthermore, the time needed to develop an application is greatlyreduced.

[0008] According to the present invention as set forth in claim 1, aconnector means is provided for automatically converting selectedcomponents with each other. In accordance with a preferred embodiment asspecified in claim 2, said selected components are—in a precedingstep—also automatically connected to said connector means. It is to benoted that the preferred embodiments of claims 15, 16, and 18 to 20 arenot limited to the embodiment of claim 2.

[0009] According to a preferred embodiment of the present invention, aconnector means is provided for automatic connection of the components.Whenever a component is added to or removed from the application itcalls the connector means. In the case of adding it to the application,it will be registered, which makes it known and accessible for theapplication, i.e., it becomes part of the application. By removing itfrom the application, it will be unregistered and can no longer be used.The adding of a component can be done by the user, for example, bydouble clicking the icon of the component, and therefore placing it on a“form”. It can also be invoked internally by other components by callinga generation operation. Removing a component can similarly be done bythe user by deleting the component from the form, or by other componentsby calling a delete operation of the component.

[0010] The generation of the connections is realized by a special set ofmethods and data structures. In the following, this set of data togetherwith the corresponding methods is called connector. The connector can,for example, be implemented as a class that generates one instance whenit is used. Each component has access to the connector. This can beimplemented by linking (in the sense of compiler terminology) theconnector to each component. When the connector is activated for thefirst time—during a particular session—it initializes its datastructures, and loads additional information for the connection process,if available. Further activations of the connector, during a particularsession, will not re-initialize it, but extend or modify its status. Theconnector holds references to all registered components. Besides this,it holds all connections between the components and all the information(for execution, communication, the load/save mechanism, etc.) derivedfrom this.

[0011] The connector divides the registered components into two groups:Those that can be connected correctly to other components, and thosethat cannot be connected. The case that a component cannot be connectedarises, for example, if the user selects only part of the componentsthat are required for a meaningful application.

[0012] One example for this is if the user selects an image processingcomponent but no component that provides the image data. Because thereare no restrictions to the order of selecting components this case willfrequently happen when setting up a new application.

[0013] After the design or user mode ends, the connector will beunloaded. Before the connector is unloaded, it can save informationabout the current components and their connections. This information canbe used when the application is invoked the next time, either in designor user mode.

[0014] Now, the addition of a component to the image processingapplication is described. When a component is added to an application bythe user, for example by double-clicking its icon, an initializationmethod of the component is executed. During this initialization, theregistration method of the connector is called. By calling this method,a reference or pointer to the component and/or its description is passedto the connector. Based on this information, the connection process willbe accomplished.

[0015] When registering a component two cases have to be distinguished.If it is possible to connect the new component to the components of theapplication, the necessary connections are configured and the componentbecomes part of the application. The integration of the new componentcan require a reconfiguration of the connections established so far.After the integration of the new component, the list of not yetconnected components is revised. This means that for each component ofthis list the registration process is executed anew (with the exceptionthat in the case it still cannot be connected it will not be added tothe list again). In the case that a component cannot be connected toother components, it is put into the list of not (yet) connectedcomponents.

[0016] Connecting a component comprises, but is not limited to, that acomponent can have (i) a reference, pointer, or a similar mechanism toits predecessors, i.e., the components which provides input data, (ii) areference, pointer, or a similar mechanism to its successors, i.e., thecomponents which accept the output data for further processing, (iii) areference, pointer, or a similar mechanism to a viewer, that can displayits results, (iv) a reference, pointer, or a similar mechanism to acomponent, that controls components, (v) a reference, pointer, or asimilar mechanism to the scheduler, that accesses all components.

[0017] According to the invention, after a component is added to theapplication, and the connections are established, it can immediately beexecuted, even if the system is in design mode. This is possible,because each component can call any other component via the connector ordirectly. By this, the whole application can be executed, even if thereare no callbacks supported by the programming environments, and only theinterface of one component needs to be active at a time (like, e.g., inMicrosoft Visual Basic).

[0018] At any time the connector can inform the user about theconsistency of the application so far. Examples for this are, forexample, that components could not be connected or that importantcomponents are missing.

[0019] It will now be described how components are connected.

[0020] The problem of connecting the components can be solved by usingdifferent methods, some of which are based on optimization, informedsearch, and heuristic search. The algorithm used in the preferredembodiment of the invention is a stepwise elimination of ambiguitiesuntil definite connections can be assigned. The algorithm is based on,but not limited to, the proposed categories of components. The algorithmdescribed in the following assumes, that no previous information aboutthe specific application is available, i.e., the user generates a newapplication.

[0021] If a component selected by a user is the first to be registeredor if there are no components that form part of the application so far,the component is tested if it is a data source (input component) or not.In the case of an input component, it is used as the initial part of theapplication, having no connections so far. Otherwise, the component isput into the list of not connected components. However, if there arealready components in the application, the following method is executed.

[0022] As a first step, each input and output parameter of the selectedcomponent, which has to be connected to the other components, is checkedwith all available parameters of the components of the application,according to a set of evaluation rules. If a connection fulfills therules, it will be added to a candidate connection list. By using furtherrules, the amount of possible connections will be reduced until anunambiguous decision can be made or the process stops. In the candidateconnection list, the parameters of already connected components areincluded. The reason for this is that the new component can, forexample, be inserted into a sequence of already connected components. Anexample for this is a blob analysis, calculating numerical data, that isconnected to an output component that writes this data to the serialinterface. If a data visualization component is added to theapplication, it can be placed in between the blob analysis and theoutput component. In addition, it has to be noted that input and outputparameters can be used more than once. A frame grabber, for example, canpass the image to multiple components for image processing. Besidesthis, many components can be connected to a viewer, for example, todisplay data.

[0023] The first rule to select connections uses the category of thecomponents. Based on general rules that describe which categories can beconnected, the initial candidate connection list is determined. Rulesused in this context would be, for example, that a viewer is connectedto an input component, or an image inspection component is connected toa viewer. These global rules are available for the connector only andare typically not provided by the components. They represent generalknowledge about components and applications. The rules can be loadedwhen the connector is invoked, can change during run time, and can beadapted, for example, depending on the field of application (e.g.,industry or medicine), or the user.

[0024] In the next step, the data types of the input and outputparameters are examined. All connection candidates that do not fulfillgiven type rules are removed from the list. Examples for these rulesare, that a weaker type can be connected to a stronger type (e.g., aninteger can be passed to a floating point value), or that equal typescan be connected.

[0025] In the next step, the consistency of connections with theapplication are examined. This includes consistency of execution, data,and information flow. This is a global graph analysis which checks forcycles, which can lead to infinite loops or blocking. For this, eachconnection is treated as a directed arc and the components as the nodesof the corresponding graph. Using the input/output description of eachcomponent, the analysis is performed, based on standard graph analysisalgorithms (e.g., as used for the analysis of Petri Nets). Eachconnection that does not generate a consistent application will beremoved. This step guarantees that an application without problems likecycles is generated. This cannot be guaranteed with the standardapproach, where the user has to connect the components with programmingstatements.

[0026] A special case are components which implement a specificgraphical user interface of another component. These identify themselvesunambiguously by their use (i.e., the component they belong to) andtheir identification.

[0027] If an unambiguous set of connections remains, the component isadded to the application and the selected connections are established.In the case of ambiguities, further rules are applied to eliminate them.The rules can be used in combination or separately. Examples for theseare: (i) locality: The distance of the positions, where the user placedthe component on the desktop. Those, which are closer, have a higherprobability to become connected; (ii) the order of selection: Thosecomponents, which are selected one after the other have a higherprobability to be become connected; (iii) application area: Theprobability of being connected is based on the application area; (iv)user: The system asks the user which connection is preferred; (v) userprofile: The system uses information about the decisions the user madein former sessions (learning).

[0028] If ambiguities remain, the component is added to the list ofnon-connected components.

[0029] A component can also be removed from an application. This can,for example, be done by the user by deleting the component. In this caseall connections to and from this component are removed. After this, theremaining components might have to be reconfigured. In the case, thatthe removed component provided input for other components, these arereconnected, if possible.

[0030] Once the user has developed an application, and the componentsare connected, the information about the connections can be stored bythe connector. This information can preferably be reloaded when theapplication is reactivated, either in design or user mode. By this,ambiguities can be minimized, even if the application is modified. Fromthis, also general rules for the configuration can automatically belearned by the system. This can be accomplished using the learningmethods known from artificial intelligence.

[0031] In view of the above, the following limitations of the knownsystems are removed according to the invention: The components are to becombined by adding programming statements to the application. Theseprogramming statements must include information as to how to pass datafrom one component to the other, and as to how to specify the order inwhich the components are executed; to make use of data, e.g., regions ofinterest, the user must add data components to the application andconnect these to the other components by programming source code.

[0032] The present invention provides an architecture of object orientedcomponents which fulfill the following requirements.

[0033] Each component comprises a description that specifies itsbehavior. This description is available in a form in which it can beprocessed by the component in all its modes of operation. It can consistof—but is not limited to—information about: number of input and outputparameters; types of input and output parameters; category; name;description; assertions; and restrictions.

[0034] Furthermore, the selection of a component by the user invokes anautomatic configuration process. This process can be controlled by asingle component which has information about all other components or byany component based on the information provided by the component to beadded to the application. The automatic configuration consists of thefollowing steps:

[0035] (i) Connection of the components: Each component is connected toother components that either provide input data or require output dataof the component. This connection can either be direct (component tocomponent) or indirect (through a sequence of components).

[0036] (ii) Configuration of an execution method: The execution methoddetermines information like

[0037] the order in which the components are executed;

[0038] the method how data is passed from one component to another;

[0039] the data types that will be generated;

[0040] the method how data will be accumulated during execution; and

[0041] how long data must be stored and when it can be deleted.

[0042] The execution method comprises sub-methods which are useddepending on the operation mode of the application, the interactions ofthe user, and the input of data. Sub-methods are, for example, beinvoked by the availability of input data (e.g., acquisition triggersignals, image grabbing, or other input channels). In this case, allcomponents that require this input data and their dependent components(i.e., components that require results of the component) will beexecuted. Furthermore, sub-methods are be invoked by the userinteraction with the input data (e.g., the image). In this case, allcomponents that can react on this kind of user interaction will beexecuted with the data which was manipulated together with theinformation of the interaction (e.g., mouse coordinates). A furtherexample is the changes to the parameters of a component by the user. Inthis case the component—together with all its dependent components—isexecuted again.

[0043] Apart from this, the execution method preferably changesautomatically depending on the operation mode. For example, in aconfiguration mode the calibration components will be executed and in anapplication mode components which adapt parameters according to changesof the input data will be executed. Furthermore, components can beexecuted in parallel.

[0044] (iii) Global communication between all components: Apart from theconnection of components, which is needed to specify the execution orderand the paths to pass data, a connection between components isestablished to pass information. This communication allows actions likepassing information from one component to another (e.g., a componentdescription or its status), sending messages between components (e.g.,executing another component), displaying the results or the status ofother components, and acting as a graphical user interface for anothercomponent.

[0045] (iv) Determination of the data that must be processed, i.e., thestates that will occur in the design and user modes, the properties ofthe application, and the connections between the components. Based onthis, an automatic process is invoked to store and reload all necessarydata during design and user mode. This data comprises, but is notlimited to, data which represents the look and feel of the userinterface, the parameters of the components, or regions of interest.

[0046] The configuration process described above has the followingproperties. It can be executed repeatedly (reconfiguration), e.g., byintroducing additional components to the application or by changing thestatus of the application. A change of the status can be invoked fromthe outside (by the user or by input of data) or internally by thesystem. Furthermore, both configuration and reconfiguration can beperformed during design time and during program execution.

[0047] The following advantageous features are associated with thepresent invention:

[0048] No programming is required. This saves time and enables lessskilled users to develop applications.

[0049] The execution method is automatically determined. This saves timeand guarantees that no programming errors are introduced. Besides this,different execution methods can be used without extra programmingeffort. The multiple execution methods make the use of the applicationvery intuitive, because the application automatically supports astandard execution method but can switch automatically to methods thathandle user interactions. Furthermore, the system can inform the userwhether a meaningful set of components has been selected, i.e., whetherthey can be connected to an application, or whether important componentsare missing.

[0050] The components of the invention automatically generate and managethe data that is processed. This saves time, guarantees that all neededdata is available, and that no programming errors are introduced. Inaddition, it is very easy to change the number of used data (likeregions of interest) because no reprogramming is required.

[0051] Data passing between the components is provided automatically.This saves time and guarantees that no programming errors areintroduced.

[0052] Components can act as graphical user interfaces for othercomponents. This enables an easy generation of those parts of thegraphical user interface that are required.

[0053] A component can have multiple graphical user interfaces. Thisenables the user to develop an application, where the graphical userinterface can be adapted optimally to the needs of the user, and as anextension to this, the application can dynamically change the graphicaluser interface to adapt itself to the requirements of its use.

[0054] The status of design and user mode are saved (and reloaded) withthe same mechanisms. This enables the user to switch between the twomodes (in both directions) without loosing state information.Furthermore, the state information can be exchanged if needed because itis stored independently of the application.

[0055] In the invention the user mode is subdivided into sub-modes. Thisallows to change the behavior of the application easily, by switching tothe appropriate sub mode. Typical modes can, e.g., be supervisor mode(e.g., full access to all features of the application), configurationmode (e.g., access to all features which are needed to adapt theapplication to a new object that has to be measured), or applicationmode (executing the application with minimal interactions and feedback). This makes the application much more adequate for therequirements of the industry.

[0056] In the invention all components can be executed simultaneously indesign mode. This allows a much easier testing of the application.Besides this, the confusing distinction between design and user mode isalmost eliminated.

[0057] Furthermore, this allows to use fewer windows to display data: Ifonly one component can be executed at a time, an additional display isnecessary to display, e.g., the image and overlay the results, to checkif the selected parameters are correct. In the invention, one displaycan be used for all components. This saves space on the screen andgenerates a much more consistent graphical user interface.

[0058] Each component contains a description that is sufficient to makefull use of all features of the component. This is the basis for theiruse without the need of programming.

[0059] Each component automatically has access to all other componentsof the application. This allows, for example, to easily control thebehavior of the application.

[0060] The invention uses data elements that combine the raw data with adescription. This allows a much more flexible use of the data. Forexample, data passing to other components which need this data can berealized without having methods implemented for this specific data type.

[0061] In the following, the present invention will be described in moredetail. Additionally, preferred embodiments are described which can beused in conjunction with the proposed system architecture.

[0062] Execution Modes

[0063] Standard systems based on object components distinguish betweentwo operation modes: Design mode and user (execution) mode. In thedesign mode, the components are selected, programmed, and configured bythe user. A (single) selected component can also be executed in designmode. Only in user mode, the complete application is executed. Thepresent invention provides the following preferred improvements:

[0064] During the design mode, not only single components but the fullapplication (all components) can be executed. This gives the userimmediate feedback on the configuration process and therefore saves timeduring the development process.

[0065] The standard user mode is subdivided into multiple sub-modes.Depending on the sub-mode, the application changes its behavior and/orgraphical user interface. Examples for user modes would be aconfiguration mode where parameters can be adapted, and application modewhere only restricted changes of the application are allowed. Thesesub-modes can also be used, e.g., to automatically invoke calibrationcomponents in the configuration mode or adapt the user interfacedepending on the user level.

[0066] Automatic Save and Load Mechanism

[0067] The standard mechanism of component-based application developmentis to save the properties that are specified in the design mode andintegrate them into the application code, where they no longer can beadapted and saved in the same way. If a system wants to save and reloadvalues in the user mode, this must be implemented using a mechanismdifferent from the one used in the design mode.

[0068] According to the invention, save and load mechanism does notdistinguish between the design and the user mode. This has the followingadvantages. First, the system status can be stored and reloaded in bothmodes. Second, the mechanism is transparent for the user, i.e., he orshe does not have to worry about the mode which is currently active. Italways works in the same manner. Third, a status that is stored in thedesign mode is available in the user mode but also the other way around:If the user changes the status during execution and goes back to designmode the status will again be available. Fourth, because the status isnot compiled into the application, it can easily be exchanged. This canbe used, e.g., to offer a private status for each user.

[0069] Automatic Handling of Data Structures

[0070] The standard mechanism of handling data structures like images orregions of interest is to treat them as a component. This means that theuser must apply programming to connect these data structures with theother components. In addition, the user either must know in advance howmany data elements will be required during execution or the programmingwill become even more complex.

[0071] With the method of this invention, the components automaticallygenerate and manage these data structures. The user does not need toworry about their number, their generation, or destruction. If needed,the data can also be saved and loaded automatically by the mechanismdescribed above.

[0072] Passing Data between Components

[0073] The standard mechanisms to pass data between components have thefollowing restrictions: The data is passed unidirectional, i.e., acomponent provides data that is used by another component, but resultsare not passed back. The second point concerns data descriptions anddata handling. So far, data structures with predefined contents areused. This allows only a restricted flexibility when handling these datastructures.

[0074] In the invention data can be passed in both directions, i.e., acaller provides input data for a component. In response, a component canpass data on to a successor or send data back to the caller. A componentcan also send data to another component that is not directly connectedto it. These mechanisms allow, for example, to construct a viewcomponent which not only displays and manipulates images and the regionsof interest but also overlays the results of (all) components,independent of whether the components are connected directly or not.

[0075] The present invention uses data elements which combine the rawdata (like the image matrix) with an abstract data description. Examplesfor typical data are images, affine transformations, points, calibrationdata, strings, coordinates, features, measurements, or polygons. Data isalways attributed with a description. The description is designed in astandardized way so that each component can interpret it. Methods toimplement these descriptions can be found in the literature. Thedescription is used to enable components to access the data theyrequire. Besides this, data can be manipulated by components based onlyon this description. The data passed between components can be an(ordered) set of data elements. A component like a scheduler can, forexample, check the type of the data that is available and pass it on tocomponents that can process this data type (the information can beaccessed through the description of the components). The datadescription can also be used during the configuration process (seeconnector) to check which parameters of components are compatible withrespect to the data types.

[0076] Furthermore, the use of (ordered) sets of data leads to an easyway to aggregate data. For example, data that is provided by a changedetector, like an affine transformation, can be added to the regions ofinterest to correct their position, orientation, and size. This data canbe passed on by all components, independent of whether they need it ornot. Each component can request the data and access it if it isavailable. On the other hand, all components can pass this data on,independent of whether this specific data was known when the componentwas implemented.

[0077] Multiple Graphical User Interfaces

[0078] The standard mechanism in component-based application developmentis that a component has a given graphical user interface which can beconfigured in design mode.

[0079] According to the present invention, a component preferably hasmultiple user interfaces. These are selected depending on, e.g., theneeds of the user. This offers two possibilities for the user. In themost simple case, an interface is selected that best fits the needs of aspecific group of users. An image processing expert wants, for example,full access over all parameters. For more complex applications, multiplegraphical user interfaces must be integrated into one program. Anexperienced user will need access to the image processing parametersduring system setup. When the application is running in the “every daymode”, only one button for start and stop and a red or green “light” forthe system status might be needed. The application can, for example, begenerated by activating the different sub-modes in the design mode.Depending on the mode, the graphical user interface will be configured.In the user mode, only the sub-mode has to be activated to switchbetween the different user interfaces.

[0080] The present invention will be explained in further detail belowwhile referring to the drawings, wherein:

[0081]FIG. 1 shows a block diagram of an embodiment of the invention;

[0082]FIG. 2 shows an example after the activation of the components(lower left side);

[0083]FIG. 3 shows the example of FIG. 2 after the selection of aviewer;

[0084]FIG. 4 shows the example of FIG. 3 after the selection of framegrabber component;

[0085]FIG. 5 shows the example of FIG. 4 with the viewer with the onlineimage;

[0086]FIG. 6 shows the result of placing the zooming component onto theform of the example shown in FIG. 5;

[0087]FIG. 7 shows the application in user mode;

[0088]FIG. 8 shows the result of selecting the bar code reader; and

[0089]FIG. 9 shows the application in user mode.

[0090] The embodiment of FIG. 1 consists of a unit to input the image bymeans of the frame grabber 12 and the grab image component 1. This imageis connected to the viewer 2 which displays the image, and overlaysregions of interest and results of the image processing components.Besides this, it manages the data, and passes it on to the othercomponents. Directly connected to the viewer 2 are a calibrationcomponent 3, a change detection component 4, and a zooming component 5.The image processing part consists of two sequences. The first oneconsists of only one component, i.e., a measure component 6. The otherone is a blob finder 7, connected to a feature extraction component 8,which is connected to a feature display component 9. The results of bothimage processing parts are output by the output component 10. Aconnector 11 performs the connection control between the variousactivated components.

[0091] In this embodiment, the connector 11 is provided forautomatically establishing connections between the components selectedby the user.

[0092] Displaying Data and User Interaction

[0093] The system according to the present invention has at least onespecial component (i.e., the viewer) that has three major tasks.

[0094] The first task is to display input data like images, and todisplay the results of other components. These results are typicallyderived from the input data. Whenever a new image is grabbed, it will bedisplayed in the viewer 2. Then the image is passed to the imageprocessing components. Data they extract in the image (like edges,lines, coordinates, distances, or polygons) is passed back to the viewer2, which overlays this data on the image, with the visualizationdepending on the type of the data. The advantage of this feature is thatonly one display is required for images and results, which saves spaceon the screen. Besides this, the graphical user interface becomes moreintuitive and the information can be perceived faster, becausecorresponding data is presented together.

[0095] The second task of the viewer 2 is to control other components.Based on the descriptions of the other components used in theapplication, the viewer 2 can act as a global control unit. It accessesinformation like the component names, supported type of region ofinterest, or data types that can be processed. Based on thisinformation, it can automatically generate a graphical user interface.To control general properties of other components, the user only needsto interact with this graphical user interface. Based on theseinteractions, the viewer 2 sends messages to the correspondingcomponent.

[0096] The third task of the viewer is to provide other components withadditional data like regions of interest: Based on the type information,the viewer knows about the kind of input data that other components canprocess. This is used to generate regions of interest that fit to acomponent. These regions of interest can be manipulated in the graphicswindow of the viewer by overlaying them on the image data and modify orgenerate them using, e.g., mouse interactions. When a component isexecuted the corresponding regions of interest are combined with theimage and passed as input.

[0097] Categories of Components

[0098] Depending on the tasks and the behavior of a component, it isassigned to a category. Therefore, a category is a set of componentsthat have a similar behavior. A category is used, e.g., to generate theexecution method for the application, or to connect components. In thefollowing, examples for categories and the corresponding tasks aregiven:

[0099] Viewer: Displays input data and overlays it with regions ofinterest and results of other components.

[0100] Meta components: Display the overall structure of theapplication, display the control flow statically, and, during execution,display used resources and execution time.

[0101] Input: Input of image data and data from other sources likeserial interface or digital I/O.

[0102] Output: Output results of the process to devices like files,serial I/O, or buses.

[0103] Calibration: Calibration of the vision system and the inputdevices, like geometric calibration or radiometric calibration.

[0104] Change detection: Changes in the input data (like illumination orpositions of reference objects) are detected and the change informationis passed on to other components.

[0105] Inspection: These components are used to extract information fromthe input data like images. This can, for example, be zooming,histogram, noise level, illumination, contrast, or sharpness.

[0106] Authentication: The user who wants to use the application has toidentify himself. Based on this information, the application can denythe use, change its behavior, change the graphical user interface, orselect the user mode.

[0107] Graphical user interface: A component can simply be a graphicaluser interface that is used in combination with another component whichdoes the data processing. Such a component allows the (partial) controlof the functionality of the component. A component preferably hasmultiple user interface components which can be selected, e.g.,depending on the user, or the user mode.

[0108] Image processing: The components have iconic data as input (e.g.,images, regions, or polygons) and output iconic and/or numerical data.

[0109] Data processing: These components process general data like theresult of a feature extraction or the input of a serial interface.

[0110] Execution Methods

[0111] The execution of the application has three methods—but is notrestricted to three—which are triggered by the interactions of the useror input from other devices; (i) driven by input data (e.g., externaltrigger, image grabbing, or other input channels): In this case, allcomponents that require this data will be executed. This is done toexecute all components according to an automatically generated schedule.This sequence of executions can either be passive (triggered from theinput) or active (requesting data, e.g., from the frame grabber); (ii)user interaction with the input data (e.g., the image): In this case,all components, that can react on this kind of user interaction will beexecuted with the data, that was manipulated together with theinformation of the interaction. As an example, the user moves the mousein the graphics window of the viewer, that displays an image. Thisinteraction results in the execution of all components, that can processimages in combination with a point (e.g., a component for zooming); and(iii) the user changes the status (parameters) of a component: In thiscase the component—together with all its dependent components—isexecuted again. If the user changes, e.g., a threshold of the blobanalysis, the scheduler will execute the blob analysis, that thengenerates a new segmentation which can be overlaid on the correspondingimage in the viewer.

[0112] Before the standard execution method will be described, thepurpose and function of the scheduler is explained.

[0113] The scheduler is that part of the invention which specifies theorder of the execution, manages the data, passes the data between thecomponents, and executes the components.

[0114] There are three major ways to implement the scheduler:

[0115] The scheduler is a component which is connected to all othercomponents. By using the mechanism for component communication, proposedby the invention, e.g., execution of the components and transfer ofdata, this component implements the features of the scheduler.

[0116] The scheduler is a software module like the connector. Thismodule is connected to the components when they connect to theconnector. Based on these connections, the scheduler can realize thefeatures of the scheduler. With this implementation, the user does nothave to add a scheduler component to the application, which makes theuse of the system easier. In this case, the scheduler would be linked(in terms of compiler technology) to the connector.

[0117] The scheduler is described among the components. Thefunctionality specific for a component is part of the component. Thiscan be implemented by offering a library which is used when a componentis implemented. This approach is slightly faster than the other twobecause less communication is required. It is the implementation used ina preferred embodiment of the invention.

[0118] The standard execution method corresponds to the first of theabove described method. It is based on a set of rules that are appliedto all components in a given order. The user does not have to worryabout this configuration process.

[0119] Main loop: The scheduler executes the components in a givenorder. It starts with the components that input data (e.g., grab animage). Whenever these components provide new data, all directly andindirectly dependent components are executed. Once through the sequence,the process starts from beginning. This restart can be invoked either byan external event, like a trigger signal or by actively requesting thedata. In both cases, the system has to wait until the data is availableto restart the standard execution sequence. Whenever there areinteractions caused, for example, by the user, the system can switch toone of the two other execution methods described above. This switch canhappen, when the system reaches the end of an execution sequence. Thisensure that only consistent results are calculated.

[0120] Input of data: The first action is to grab the image or accessother external data. This input can be invoked actively, or by anexternal signal. In FIG. 1, component 1 is executed.

[0121] Displaying: Depending on the sub-mode and used properties, theimage will be displayed. Optionally, it can be overlaid with regions ofinterests or the other (input) data. In FIG. 1, component 2 (viewer) isexecuted.

[0122] Primary vision loop: Depending on the execution sub-mode,specific preprocessing components are executed.

[0123] Inspection: These components can be used in all sub-modes. InFIG. 1, this corresponds to component 5 (zooming).

[0124] Calibration: These components are used in the configuration mode.When they are executed, they return data that must be stored to be usedlater in the application mode. It will be added to the input data toenable the vision components to calibrate the results and to increasetheir accuracy. In FIG. 1, this corresponds to component 3.

[0125] Change detection: In the application mode, these componentsreturn data that will be added to the image by the display to enable thevision components to compensate these changes. In FIG. 1, thiscorresponds to component 4.

[0126] Vision loop: The scheduler combines the data from the primarycomponents (e.g., calibration data 3 or change detection 4) with theimage. Then a loop for all connected vision components is started. Thismeans that each directly connected vision component is provided withinformation and then started. As a consequence, all successors of thecomponent will also be executed. For each directly connected visioncomponent the scheduler has a list of regions of interest. This list isalso added to the image data.

[0127] Execution: The compound image data is passed to the visioncomponent and then executed.

[0128] Successor: In the case that a component has a successor, it willpass its input data, combined with its results, on to the successor, andexecutes it.

[0129] If necessary, the component can modify the input data before itis passed on to the successor. This process is repeated until allsuccessors are executed.

[0130] Vision results: Whenever a component calculates results, they canbe passed back to the viewer, where they can be overlaid with the imageand other data displayed so far. This way of displaying results worksindependently of the component being directly connected or not.

[0131] Non-iconic results: Data like areas, distances, or text is alsopassed back to the scheduler. Here, it can be collected until allresults are available.

[0132] Control messages: It might be necessary that a component controlshardware, e.g., a frame grabber. In this case, the component can send amessage that generates the desired action back to the scheduler.

[0133] In FIG. 1, two sequences are available. The first sequenceconsists only of the measure component 6. This is executed first. Theresults of the measurement are passed back to the viewer 2. The secondsequence consists of components 7, 8, and 9. They are executed in thisorder. The results are passed back to the viewer 2 by the last component9, i.e., feature display of the sequence.

[0134] Output: After all vision components are executed the result datais available at the viewer 2. Therefore, it is passed on to allconnected output components. In FIG. 1, one output component 10 isconnected.

[0135] With an execution caused by interaction, there are two ways, bywhich the user can influence the execution method. According to thefirst way, the user interacts with the data visible in the viewer: Incase the user points or clicks on input data like an image, the viewercollects these events as long as the execution is in the standardexecution method. When the scheduler finishes the standard executionsequence, the interactions are examined, and, e.g., duplicateseliminated. Then, all components that can process this kind of data areexecuted with the data currently available, combined with the mousepositions and button press data. After that, the scheduler can proceedwith the standard execution method. In FIG. 1, one component that canprocess mouse interaction is connected, namely zooming component 5. Itwill be executed in this case.

[0136] Alternatively, the user interacts with any component, forexample, by changing image processing parameters. Whenever this happens,the component informs the scheduler that the component has to bere-executed. The scheduler holds a list of the components that have sentthis request. When the standard execution sequence is finished, thislist is processed. This means that all sub-sequences that contain acomponent of the list are re-executed. As an example, the user interactswith the blob finder 7 in FIG. 1. This results in a re-execution of thecomponents 7, 8, and 9.

[0137] Depending on the application, the execution methods describedabove are only preferably used. They are not necessary, for example, forapplications that are time critical. Furthermore, it is possible toprocess the components with new data, for example, from the framegrabber, instead of using the old data.

[0138] Now, implementation details are described. To avoid errors whenprogramming a new component and to ease the task of development a newcomponent, the following rules can be applied. First, a root class,which implements basic data structures and methods that are common toall components, is used. This class is used as an interface for allcomponents. This guarantees that all components have the same basicfunctionality. The root class can implement all the functionality thatare common to the components of the invention. Besides this, one classfor each category that is derived from the root class is implemented.These classes implement data structures and methods that are specificfor the categories. These classes are used as an interface for thecomponents belonging to a specific category. Finally, each component isderived from the class of one category. This guarantees that a componentimplements all features that components have in common and besides thishave the specific features of the category.

[0139] With respect to load/save in design and user mode, the followingshould be noted: When a component is invoked for the first time during asession—independent if this happens in design or user mode—the state ofthis component, in the context of the application, is loaded. If nostate information for the component in combination with the applicationis available, the default value will be returned.

[0140] When the component is destroyed, for example, at the end of theexecution or at the end of the design time, its status is saved. Thedata is stored in the context of the application. This means that thestate information of an application is defined as the set of states ofthe components belonging to it. The state information of an applicationcan, for example, be stored in a file or the registry of the operatingsystem.

EXAMPLE

[0141] The following example describes the development of a typicalapplication that grabs the images from the camera, displays these imagesin a viewer, checks for bar codes in the image, reads the codes, andpasses the results to a serial interface. In addition, an online zoomingis added to inspect the images.

[0142] Starting the Development Environment

[0143] The examples assumes that Microsoft Visual Basic is used todevelop the application. Therefore, this environment is started by theuser and “Standard-EXE” is selected. After this, the user selects thedialog “Components” in the menu “Project” to activate the components ofthe invention. After the activation for each component, an icon isdisplayed in the component selection. Now the user can start thedevelopment of the application. FIG. 2 shows Visual Basic after theactivation of the components (lower left side). The area in the middle“(Form 1)” is used to place the components of the application.

[0144] Selection of Viewer

[0145] As a first step, the user double-clicks the icon of the viewer.This generates an instance of this component which is automaticallyplaced into the form of the application that is being developed (seeFIG. 3). The size, the position, and other properties of the viewer canbe changed by the user. This can be done by manipulating the viewer withthe mouse (e.g., position and size) or by changing the propertiesdirectly (e.g., by using the property page at the lower right side ofthe window).

[0146] When the viewer 2 is placed on the form (“Form 1”), theinitialization routine of the component is executed and after this thecomponent is drawn on the form. During the initialization routine, thecomponent calls the registration routine of the connector 11. It passesa reference to itself as a parameter to the connector. Because theconnector 11 is called for the first time, it initializes its datastructures. Using the reference to the component, the connector canaccess the description of the component. It first checks whether thecomponent is an input component. Because the viewer 2 is not an inputcomponent, it will be put into the list of not connected components. Thelist of connected components remains empty. Therefore, the single viewercomponent does not implement an application.

[0147] Whenever the user interacts with the component (by using themouse or by changing the properties) the corresponding routine of thecomponent is executed.

[0148] Selection of Frame Grabber Component

[0149] As the next step of this example, the user selects the framegrabber component 12 by double clicking its icon. As a result, thecomponent is placed on the form. Like with the viewer 2, the user adaptsthe size, the position and other properties interactively. The result ofthese actions can be seen in FIG. 4.

[0150] When the frame grabber component 1 is placed on the form it callsthe connector 11 (like the viewer 2). By passing a reference to itselfto the connector, the automatic connection process is invoked. Theconnector first checks for the category of the component. Because thiscategory is “input”, the frame grabber 1 becomes the first component ofthe application. So far the component has no connections. As the secondstep, the connector reexamines the components in the list of notconnected components. Here, only one component is available: the viewer.Now all rules for connecting parameters are tested. In this case theframe grabber is connected to the viewer because:

[0151] input components can be connected to view components,

[0152] the output type of the frame grabber is “image”, and

[0153] the input type of the viewer is “image”.

[0154] Therefore, the frame grabber 12 is connected to the viewer 2 andalso the viewer 2 is connected to the frame grabber 12 This means thatboth components get a reference to the other component. Using thisreference, they can exchange data and call routines of each other.

[0155] Execution in Design Mode

[0156] In order to execute the application in the design mode the userclicks with the right mouse button on the frame grabber. This creates apop-up menu where the last entry (“Edit”) is selected. This enables theuser to use the graphical entities (buttons, option menus, etc.) of thecomponent. Using these, the user selects the appropriate frame grabberand its parameters and switches it online.

[0157] When the user activates the component using the pop-up menu, allits data structures and routines become active, i.e., everything can beexecuted when it is invoked by interactions of the user. When the userchanges the parameters of the frame grabber it accesses the signal fromthe camera. This generates an image in the frame grabber component.Because the frame grabber is connected to the viewer (as a data source)it will pass this image to the viewer. This activates the viewer and theviewer displays the image. As long as the frame grabber remains active,it generates images, which are passed on to the viewer and displayedthere. The effect for the user is the online image in the viewer can beseen and the frame grabber parameters are edited in the frame grabbercomponent. FIG. 5 shows the viewer with the online image.

[0158] Selection of the Online Zooming Component

[0159] To inspect the quality of the image, the user adds an onlinezooming component 5. The actions are the same as those of the previoustwo components.

[0160] The zooming component is then connected to the viewer because thecategory of the zooming component is “inspection”, inspection componentscan be connected to view components, the input data type of the zoomingcomponent is “image”, and the viewer provides the output data type“image”.

[0161] The user then activates the viewer. When s the mouse is moved inthe display, the zooming component displays the part of the image inwhich the mouse is located. As regards the system, two actions areimportant for the online zooming: Whenever an image is passed to theviewer (from the frame grabber), it will automatically be passed on tothe zooming component. Furthermore, if the user interacts with the mousein the display of the viewer, the zooming component will be called withthe image and the mouse coordinates.

[0162]FIG. 6 shows the result of placing the zooming component onto theform. Besides this, the effect of moving the mouse in the viewer isvisible.

[0163] Execution in User Mode

[0164]FIG. 7 shows the application in user mode. The user presses therun button of the Visual Basic environment to execute the application inuser mode. This starts the program as a stand-alone application. Theuser can interact with it in the normal way (editing frame grabberparameters, displaying the image in the online zooming component).Finally, the user stops the application and returns to the design mode.Changes applied to the application in the user are again available inthe design mode.

[0165] When the user presses the run button, all components of theapplication are destroyed. Before this happens they save their stateinformation (like parameter settings) in a special state file. Afterthis, Visual Basic creates an executable file that is executed. Whenthis happens, all components are created anew and are placed at theircorresponding positions. After this, the components read their stateinformation from the file and reconfigure themselves accordingly. Then,all components call the connector to generate the same connections thatwere used in the design mode. Finally, the application waits for theinput of the user. When the user switches the frame grabber on, thesystem runs with the default execution mechanism: The frame grabbercomponent waits for an image, passes it on to the viewer, which displaysit and passes it on to the zooming component. Whenever the user movesthe mouse in the display of the viewer, this generates an extra call tothe zooming with the mouse coordinate.

[0166] When the user stops the application, all components aredestroyed. Before that, the components save their state information inthe state file.

[0167] When Visual Basic returns to the design mode, the components aregenerated anew, their state information is reloaded and the componentsare re-connected.

[0168]FIG. 7 shows the application in user mode.

[0169] Selection of Bar Code Component

[0170] To complete the application, the user adds a bar code reader toit. FIG. 8 shows the result of selecting the bar code reader. When thecomponent is selected and generated, it is connected to the application.The following rules are used:

[0171] The bar code reader is a vision component

[0172] Vision components can be connected to a viewer

[0173] The input type of a bar code reader is “image”

[0174] The viewer has an output for images

[0175] Defining a Region of Interest

[0176] Finally, a graphical component is added to the application tocreate and edit regions of interest. The region of interest component isconnected to the viewer, because:

[0177] The category of the component is “graphical user interfacecomponent” and

[0178] it can be connected to the viewer

[0179] After the component is connected, the viewer provides informationabout all components that depend on the viewer to the new component.From this information, the following is selected:

[0180] Names of the connected components

[0181] Type of the regions of interest that are supported

[0182] Maximum number of regions of interest

[0183] Using this information, the dialog is configured accordingly.

[0184] Execution in User Mode

[0185] In order to execute the application in user mode, the userpresses the run button. In user mode, the region of interest for thebarcode reader is specified. By doing this, the application isautomatically executed and the bar code reader is executed with thespecified region of interest. When the user changes parameters of thebarcode reader, the component will be re-executed with the newparameters. When the frame grabber provides new images, the viewerdisplays the image and executes the zooming and the bar code component.

[0186] When the status changes from design to user mode—and viceversa—the same mechanism described above is executed. When the userselects a component in the region of interest component, the types ofregions of interest that are supported by the component can be selected.By choosing one of these, the user can interactively generate a regionof interest in the viewer. This region of interest is stored with theviewer, together with the information about the component it belongs to.Whenever the component is executed, in the given example the bar codereader, the region of interest will be combined with the imageautomatically when the image is passed to the component.

[0187]FIG. 9 shows the application in user mode. In the region ofinterest component (left center) the component “1D Barcode” and arectangular region of interest type is selected. The generated region ofinterest is visible as, e.g., a red rectangle in the viewer. In the barcode component (upper left) the result of the decoding is displayed. Theregion corresponding to the code is displayed as, e.g., a blue border inthe viewer. This is accomplished by passing the segmentation result backto the viewer and overlaying it to the image.

[0188] Saving the Application

[0189] When the user returns to design mode, he/she can generate a standalone application using standard Visual Basic mechanisms. This generatesan “exe”-file and a file that contains the state information.

1. A method for developing image processing applications, comprising thesteps of: (a) providing a plurality of image processing components (1 to10); (b) providing a connector means (11) for establishing connectionsbetween components selected from said plurality of components by a user;and (c) automatically connecting said selected components with eachother by using the connector means (11) to generate said imageprocessing application.
 2. The method of claim 1, further comprisingbetween steps (b) and (c) the step of automatically connecting saidselected components to said connector means (11).
 3. The methodaccording to claim 1 or 2, wherein in step (c) the automatic connectionof the selected components is based on information provided by therespective components.
 4. The method according to claim 1, 2 or 3,wherein the connector means (11) initializes its data structure whenbeing activated for the first time.
 5. The method according to any ofclaims 1 to 4, wherein the connector means (11) registers said selectedcomponents (1 to 10).
 6. The method according to claim 5, wherein saidselected components are registered by said connector means (11) in atleast two groups dependent on the ability of each selected component ofbeing connectable to other selected components or not.
 7. The methodaccording to any of claims 1 to 6, wherein each selected componentpasses a reference or pointer to the connector means (11) and/or itsdescription to said connector means (11).
 8. The method according to anyof claims 1 to 7, wherein the first selected component is used as theinitial component if it is an input component, or otherwise added to alist of not-connected components.
 9. The method of claim 8, furthercomprising the steps of: checking each subsequent component withavailable parameters of the prior selected components according to a setof evaluation rules, and adding it to a candidate connection list if aconnection fulfills said evaluation rules; examining data types of inputand output parameters of said selected component; examining consistencyof connections with the image processing application; adding thecomponent to the image processing application; and establishing theselected connections.
 10. The method of claim 9, wherein said evaluationrules comprise a graph analysis to determine consistency and to avoiddeadlocks during application execution.
 11. The method according toclaim 9 or 10, wherein said evaluation rules utilize the category ofcomponents, locality, order of selection, application area, user, anduser profile for establishing a connection.
 12. The method according toany of claims 1 to 11, wherein said connector means (11) savesinformation about the current components and their connections beforebeing unloaded.
 13. The method according to claim 12, wherein saidinformation is reloadable when reactivating the image processingapplication.
 14. The method according to any of claims 1 to 13, whereinat least one component (1 to 10) comprises a description specifying itsbehavior, said description comprising information selected from thegroup comprising number of input and output parameters, types of inputand output parameters, category, name, description, assertions, andrestrictions.
 15. The method according to any of claims 1 to 14, whereinsaid selected components are simultaneously executable in the designmode.
 16. The method according to any of claims 1 to 15, wherein thesystem status is storable and reloadable in design mode and/or usermode, whereas whatever was stored in design mode is reloadable in usermode and vice versa.
 17. The method according to any of claims 1 to 16,wherein data structures of the image processing application areautomatically generated and managed.
 18. The method according to any ofclaims 1 to 17, wherein data is passed between components in bothdirections.
 19. The method according to any of claims 1 to 18, wherein acomponent sends data to a component not directly connected therewith.20. The method according to any of claims 1 to 19, wherein at least onecomponent has multiple user interfaces.
 21. The method according to anyof claims 1 to 20, further comprising the steps of: connecting theselected components (1 to 10) for passing data; configuring an executionmethod; establishing global communication between all selectedcomponents for passing information; and determining data to beprocessed.
 22. A computer readable program product for performing amethod for developing image processing applications as claimed in claims1 to
 21. 23. A system for developing image processing applications,comprising: a plurality of image processing components (1 to 10); andconnector means (11) for establishing connections between componentsselected from said plurality of components by a user, wherein saidconnector means automatically connects said selected components.